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BACKGROUND 


The Department of Fish, 
Wildlife & Parks (FWP) 
manages the state’s hunting 
licensing and permitting 
functions, including 

13 special permit drawings. 
These limited and often 
highly sought-after 
opportunities are obtained 
through random drawings 
when the number of 
applications exceed quota set 
by the commission. Hunters 
and anglers may increase 
their chances of selection by 
purchasing a species-specific 
bonus point. Bonus points 
increase the number of 
chances a hunter has in the 
drawing. However, bonus 
points do not guarantee 
selection, as the drawings are 
random. 
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ANALYZING BONUS POINT ACCURACY 


FWP accurately calculates and consistently applies 
bonus points as required by law and rule. However, we 
identified improvements in data quality protection, 

use of historical data, and communications needed to 
ensure ongoing success. The improvements will protect 
the integrity of bonus point processes and offer clear 
descriptions of bonus points to the public and within the 
department. 


What We Did 


We designed this audit to determine if FWP accurately calculates 
and consistently applies bonus points as required by law 

and rule. The methods used to answer this question include 
reviewing the computer system code related to bonus point 
procedures. We also reviewed law and rule related to bonus 
points. We then compared those requirements to bonus point 
behavior in drawing data from the 104 drawings using bonus 
points occurring from 2014-2021. We simulated millions of 
draws to determine if the results for bonus point holders 

aligned with expected probabilities. We also conducted several 
statistical tests to determine if the random number generator 
was operating as intended. We reviewed the controls FWP has in 
place for both the bonus point data and the system processing 
the draws. In addition, as a successful public-facing program 
must be understood by the public, we reviewed materials 
available to hunters and the public regarding bonus points 

and interviewed hunters and stakeholder groups about their 
understanding of bonus points. 


What We Found 


All tests of FWP’s process revealed bonus points behaving 

as law and rule require. However, to ensure accurate and 
consistent drawings continue, we found improved process 
controls are necessary. Specifically, to reduce the risk of 

bonus point errors, the department needs to increase controls 
related to the Automated Licensing System (ALS), the computer 
system housing all bonus point data and processing the permit 
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drawings. We found an increased risk 

in the following areas: manual data 
changes, verifying accrual of points, and 
computer code descriptions. 


"All tests of FWP's 
process revealed bonus 


We also found the clarity and consistency 


points behaving as law 
of communications regarding the bonus and rule require." 
point program need improvement. We 


found varying knowledge levels among FWP staff regarding important bonus point 
facts. In addition, information available to the public regarding bonus points is 
confusing. Some hunters are unclear about the calculation process and how bonus 
points increase their chance of selection in a permit drawing. 


Bonus Points Widely Used 

In 2022, FWP received more than 314,000 applications for license/permits in 
draws with bonus points. Most drawings discussed throughout this report are for a 
license and permit. However, since bonus points are usually related to permits, we 
will use the term "permit" to refer to the license/permit combination. As hunters 
and anglers commonly apply for more than one species, we estimate the number 
of individual applicants using bonus points in 2022 to be between 275,000 and 
300,000. Table 1 illustrates more details regarding bonus point use in 2022. 


Table 1 


The Number of Bonus Points Used in 2022 Drawings Was 936,658 With a 
Value of More Than $4,600,000 


Species Residency Applicants Bonus Points 
Resident 62,505 70,212 
Non-Resident 11,463 16,504 
Resident 44,265 92,848 
Non-Resident 6,292 6,278 
Resident 83,454 164,430 
Non-Resident 9,921 14,969 
Resident 31,058 172,764 
Non-Resident 3,337 23,040 
Resident 26,731 177,568 


Antelope 


Deer 


Elk 


Moose 


Sheep 


Non-Resident 


7,483 


66,979 


Goat 


Resident 
Non-Resident 


18,497 
3,532 


101,461 
25,829 


Mountain Lion 


Resident 


412 


967 


Non-Resident 


56 


191 


Paddlefish 


Resident 


1,392 


Non-Resident 


156 


31 


Swan 


Resident 


981 


1,068 


Non-Resident 


166 


127 


Totals 


Resident 


271,835 


782,710 


Non-Resident 


42,406 


153,948 


Grand Totals 


314,241 


936,658 


Source: Compiled by the Legislative Audit Division from department records. 


ALS Implements Bonus Point Life Cycle 


To further explain the required behavior of bonus points, Figure 1 describes their life 
cycle from a hunter's or angler's perspective during one season's draw process. 


As shown in Figure 1, the bonus point life cycle has many interrelated steps. We 
found these and other processes used by FWP to implement the bonus point 
program are based on requirements in law and rule. 


The requirements include: 


* 


An applicant may 
purchase only one bonus 
point per species per 
year. 


Bonus points may be 
purchased at time of 
application or between 
July 1 and September 30. 


The number of chances 
an applicant has in 

a drawing is 1-- (their 
bonus points)". 


Bonus points accumulate 
over time. Those 
purchased in previous 
years can apply the next 
time the hunter or angler 
applies for a permit. 


If an applicant is 
successful in a permit 
drawing, their bonus 
points for that species 
drops to zero. 


A hunter loses all bonus 
points for every species 
when their hunting 
privileges are suspended 
or revoked. 


Figure 1 


Apply for Special Permit 


All applicants receive one chance to win a special permit. 


Purchase Bonus Points 

Bonus points can be purchased for $2 or $20 for non-residents. 
Bonus point holders receive one chance PLUS additional chances 
based on their bonus points. 


Assign Random Numbers 


Each chance is assigned a random number. A low number is 
considered good. 


Place Numbers in the Draw 

For bonus point holders, the lowest of their random numbers 

are entered into the draw. Those without bonus points will have their 
single number put into the draw. 


Sort Numbers 
Applicants are sorted by their random number with the lowest 
number on top. 


Select Successful Applicants 
Permit winners are chosen from the lowest to the highest number 
until the available permits are gone. 
Source: Created by the Legislative Audit Division from department 
records, law, and rule. 


Numerous and Diverse Tests 
of ALS 


Our work aimed to assess 
the ability of FWP's system 


to meet legal requirements à 
related to bonus points. - | "uf meets regm 
Figure 2 describes where PH C 
LAD ran tests to check if the 
system was operating as 


intended. 


Systems Tested 


When examining the system We tested the computer system's assignment of chances and 


code, we determined found all applicants received the correct number of chances. 
the overall procedure for 

performing the draw was RNG Tested 
the same across all species. We tested FWP's random number generator and found no 
Nevertheless, we tested a predictable patterns. 


variety of draws, such as 
Me ode Tested 

mountain lion, moose, sheep, y 

and others, to assess if the We tested the system code and found it performs these 

processes were working as procedures correctly: 

intended, and we found they Win Rates Tested 

were. Table 2 provides more 

detail regarding the specific We tested applicant win rates and found they align with theoretical 


tests we ran related to FWP’s a la 
bonus point process. Source: Compiled by the Legislative Audit Division from department 
records, law, and rule. 


Table 2 
Diverse Tests Found Bonus Point Behavior as Required by Law and Rule 


Key Question Data Tested Tests Performed Result 


Do applicants receive Compared results of calculations 
the correct number of required by law to applicant's 
chances based on their actual number of chances to 
bonus points? determine if they are the same. 


Monobit Test 
Block Frequency Test No predictable patterns 
Runs Test appear in random 
Fourier Analysis numbers 
Poker/Five Value Test 


Applicants receive 
correct number of 
chances 


2022 moose drawing data 


Does FWP's random 
number generator Set of 1 million random 

produce predictable numbers generated by FWP 
patterns in numbers? 


The 2022 mountain lion, We statistically compared 
moose, sheep, and other simulation results with theoretical 
species drawings were probabilities using a 99 percent 
simulated up to 1 million times. | confidence level. 


Do applicant win rates 
align with theoretical 
expectations? 


Applicant win rates 
align with theoretical 
expectations 


Does FWP's computer 
System correctly 
perform bonus draw 
procedures? 


We determined if the required FWP computer system 
FWP's Computer System Code | procedures would result from correctly performs all 
how the system was coded. draw procedures 


Source: Compiled by the Legislative Audit Division. 


Bonus Points Provide Additional Chances 


The purpose of the bonus point program is to benefit applicants purchasing and 
accumulating bonus points. Successful applicants are chosen from the lowest to the 
highest assigned random number. Consequently, all applicants want to be assigned 
a low random number. If an applicant has bonus points and chooses to use them, 
they receive additional chances for a low number. The number of additional chances 
is based on their number of bonus points and the formula required in law, 1+(bonus 
points)?. To further describe the relationship between the number of bonus points 
and chances to be assigned a low random number, Figure 3 compares three 
persons with different numbers of bonus points. 


Figure 3 
Purchasing and Accumulating Bonus Points 
Person A Person B Person C 
0 accumulated points 2 accumulated points 9 accumulated points 
+ + + 
0 purchased points 1 purchased point 1 purchased point 
0 total points 3 total points 10 total points 
1 chance 10 chances 101 chances 
1 + (0)? 1 + (3)? 1 + (10)? 


Source: Compiled by the Legislative Audit Division from department records, law, and rule. 


Our Conclusion 


All tests of FWP’s process confirmed bonus points behave as law and rule require. 
We conducted tests on FWP's bonus point processes from initial application 

to selection of successful applicants. The number of chances an applicant has 

is determined correctly based on their bonus point total. The random number 
generator operates appropriately, and the simulated win rates match theoretical 
expectations. The computer system code conducts the draw procedure with bonus 
points as intended and selects winners based on a standard sorting process. FWP’s 
bonus point process is accurate, consistent, and in accordance with law and rule. 
Hunters, anglers, and policymakers can depend on the accurate selection of permit 
draw winners. 


More Work on Controls, Risk Identification, and Communications Needed 


While we determined the bonus point process is currently working as intended, 
further effort is needed to ensure accurate and consistent drawings continue. We 
make three recommendations related to the specific areas we found needing 
improvement: data quality and process protection, identifying accrual risk, and 
communications. Specifically, we reviewed manual changes to bonus point data, 
computer code descriptions, the use of historical bonus point data, and the clarity 
and consistency of communication regarding the bonus point program. 


Additional Protection Against Errors Needed 


To protect the quality of data and processes, the federal Standards for Internal 
Control, Montana Operating Manual, and industry standards state complex computer 
systems need the responsibilities of each position interacting with the system to 

be distinctly identified and well understood by the individual in the position and 
management. Documentation of this material assists management in designing 
overall protections for the system and mitigates the risk of this knowledge being 
limited to a few staff. Due to other priorities in the large permitting and licensing 
programs, FWP has not taken steps to formalize needed data quality and system 
controls such as reconciliation of data change requests and code descriptions. 


Bonus Point Eliminations Lack Centralized Tracking Method 


A risk to ALS data is the informal process for removing bonus points due to court 
action. When a hunter loses hunting privileges due to enforcement action, they 

also lose all accumulated bonus points for all species. FWP's process for tracking 
this action and removing bonus points involves sending emails from enforcement 
administration to three licensing staff. One makes the change manually and keeps 
the email on their computer as a record. No policy requires a centralized record 

of the requests, nor is there a way to periodically reconcile enforcement requests 
with ALS changes. Data quality in the ALS would benefit from formalized tracking of 
enforcement requests. 


Code Descriptions Needed 


We determined the ALS system code currently works as intended to conduct draws, 
but further effort is needed to ensure continued functionality. When reviewing 

the ALS system code with FWP's Technology Services Division, we found the code 
contains limited descriptions of its procedures and the relationships between 
procedures. We also found legacy code no longer applicable to draws but kept in 
case a law or rule reverts back to a previous version. Accurate descriptions of code 
are an industry best practice. Without description, staff unfamiliar with the code 
and its history may incorrectly modify the coded procedures. The department is 
migrating the draw procedure to a new system and not relying on a specific coding 
language. As part of this process, the department will rebuild sections of the code 
and report in the new system annually. Given this migration, it does not make sense 
for the department to use resources now describing old code slated for rebuilding. 
However, the need to describe procedures and relationships in the new system 
remains. 


Decreased Protection Against Errors 


While we did not identify any errors by FWP's systems, the absence of formally 
established protections for bonus point data quality and code descriptions increases 
the risk of errors. In addition, informal controls and limited code descriptions 
increase risk in other ways such as loss of organizational knowledge. 


Recommendation #1 


We recommend the Department of Fish, Wildlife & Parks improve its bonus point 
protection structure in the Automated Licensing System by developing, documenting, 
and implementing: 


A. A process to periodically reconcile bonus point change requests from enforcement 
and 


B. A process for describing all draw code and procedures as part of the department's 
ongoing migration process. 


Risk Assessment and Management Improvements 


The federal government's Standards for Internal Control and the COBIT Governance 
and Management Objectives framework indicate an organization should identify 
and collect relevant data to enable effective computer-related risk identification, 
analysis, and reporting. Also, an organization needs to keep and analyze sufficient 
historical data to satisfy organizational and regulatory requirements. Guidance 
regarding how this can be accomplished includes using policies and processes 
compelling analysis and controlling access and modifications to historical data. 


Enhanced Controls Needed for Manual Changes 


The department currently keeps all changes made to each applicant's bonus 

points. This includes remarks for changes made to bonus point data. Routine 
changes to bonus point data auto-generate remarks explaining the change. 
However, some changes to the data must be made manually. These remarks do 

not have a consistent format, making this control not as effective as it could be. 

In addition, the system does allow a blank remark field. Although we only found 
one instance of this in our work, it increases the risk of manual changes without a 
listed reason. A consistently used format and a prohibition of blank remark fields 
would better protect the data's quality. It would also create a more useful historical 
data resource for the department because it would be easier to spot patterns and 
anomalies. ALS automatically creates another historical data resource, but the value 
of this control is greatly diminished because it is not reviewed by management. ALS 
creates a log of any manual changes. It records the user, date, and time of each 
manual change. 


Process to Identify Potential Risks in Bonus Point Behavior Needed 


We determined there are potential risks in bonus point accrual that we, and the 
department, are unable to identify. The department must calculate each applicant's 
bonus point amount for each draw by reconstructing all historical bonus point 
changes. However, the department does not have a process to consistently review 
bonus point accrual, either retroactively by checking applicants' bonus point 
amounts for each draw or by checking point changes during each draw. A review 
process would help ensure that bonus points are accruing appropriately. 


Potential Error Risk Increased 


Inconsistent manual adjustment comments, limited processes to identify accrual 
risks, and a lack of manual change review increase the potential for undetected 
errors. Our work found no evidence of undetected errors or incorrect manual 
changes. However, the potential risk of incorrect manual changes, manual changes 
for personal gain, or inaccurate bonus point accrual remains without a review of 
these processes. Any of these risks would undermine the consistency and fairness 
of bonus points and the draw procedure. 


Recommendation #2 


We recommend the Department of Fish, Wildlife & Parks increase its risk identification 
and management in the permit draw and bonus point data structures by developing, 
documenting, and implementing: 


A. Consistent formatting for manual adjustment remarks, 
B. A review process for manual adjustments, and 


C. A process to identify risk in bonus point accrual. 


Clearer Communication Needed 


The federal government and industry standards provide guidance for successful 
organizational communication. All indicate management should communicate 
information to achieve an organization's objectives. Specifically, management 
should select appropriate methods to communicate externally and internally 

based on the audience and the purpose of the information. If lost, there is no 
simple answer for how government can keep or recapture public trust. However, 
communicating simply with repeated, consistent messaging is necessary for any 
successful government communication effort. Regarding communication within the 
department, research indicates the successful exchange of information between 
organizational divisions is beneficial to the organization. It avoids internal confusion 
and mixed messages to public. 


Bonus Point Public Information Unclear 


The internal and external communication efforts for permitting draws and bonus 
points are underdeveloped. We found some information regarding bonus points, 
both online and in the hard copy regulations booklet, difficult to understand. In 
addition, we found instances when the public could receive different information 
about bonus points from different FWP divisions. Some aspects of FWP's 
communication efforts are sound. The Web page is straightforward, and it is easy 
to find information about bonus points. The regulations booklets are familiar to 
hunters. However, some of the information needs to be explained more clearly and 
thoroughly. For example, the FWP Web page directs the public to a "Bonus Point 
Drawing Statistics" report that is designed to explain the success of bonus point 
holders for a certain species and year. This public-facing report is a spreadsheet 
with ten columns and often thousands of rows of data. The data is not described 
other than the 2- to 3-word column titles. In addition, there are no instructions 
regarding how to search or generally navigate the large spreadsheets. 


Underdeveloped Communication Creates Confusion 


The underdeveloped bonus point communication results in an inaccurate 
understanding of bonus point processes in the department and among the public. 
We interviewed FWP wardens, unaware that revocation or suspension of hunting 
privileges eliminates all bonus points. This could result in inconsistent bonus point 
information being provided to the public. We also found some hunters unaware 
of how exactly bonus points increase their chances in a draw. This could create 
unrealistic expectations of hunters. 


Bonus Point Messaging Needs Attention 


The department has not often reviewed its internal nor external bonus point 
communications to determine if there are ways to improve its clarity and 
consistency. It is not focused on assessing the current information for the public 
about bonus points and determining if there is a way to improve it. While they 
run and post statistical reports regarding the use of bonus points, they have not 
assessed the ability of the public to understand the reports. 


Clear and Consistent Message Increases Stakeholder Confidence 


FWP must consider the clarity and consistency of its messages about bonus 

points, which is necessary for successful communication. Our work found confusing 
public-facing information. We also found no consistent message across all 

divisions regarding how bonus points work. In addition, FWP has communication 
professionals in the department who could provide expertise related to successful 
messaging, consistency, and distribution of the bonus point message. 


Recommendation #3 
We recommend the Department of Fish, Wildlife & Parks: 


A. Implement an on-going strategy to improve the clarity and consistency of bonus point 
information available to the public, and 


B. Distribute new bonus point information to public-facing FWP staff likely to interact 
with hunters and anglers. 
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Re: Focused Evaluation of Analyzing Bonus Point Accuracy 


Dear Mr. Maciver: 


The Department of Fish, Wildlife and Parks appreciates the opportunity to respond to the Focused 
Evaluation Analyzing Bonus Point Accuracy completed by the Legislative Audit Division. The Department 
has reviewed the recommendations set forth in this report and provided responses and corrective 
action plans for each recommendation below. 


Recommendation #1 
We recommend the Department of Fish, Wildlife & Parks improve its bonus point protection structure in 
the Automated Licensing System by developing, documenting, and implementing: 


A. Aprocess to periodically reconcile bonus points change request from enforcement and 


Concur - The Licensing Bureau and Enforcement Division staff will coordinate to develop a 
process to log individual action requests each license year. The established process will 
reconcile the deletion of bonus points in the Automated Licensing System and improve the 
bonus point protection structure. 


B. Aprocess for describing all draw code and procedures as part of the department's ongoing 
migration process. 


Concur - Technology Services Division staff will document draw codes and procedures as the 
business logic is migrated into the new system. 


Recommendation #2 
We recommend the Department of Fish, Wildlife & Parks increase its risk identification and management 
in the permit draw and bonus point data structures by developing, documenting, and implementing: 


A. Consistent formatting for manual adjustment remarks, 


Concur - The Licensing Bureau and Technology Services Division staff will coordinate in creating 
a limited selection of standard rational options to be used as justification for any manual 
adjustments to bonus points. 


B. Areview process for manual adjustments, and 


Concur-The Licensing Bureau staff will create a procedure to be used for any/all manual 
adjustments to bonus points, including reason for the adjustment and logging staff member 
identification and date and time of any changes. 


C. Aprocess to identify risk in bonus point accrual. 


Concur - The Licensing Bureau and Technology Services Division staff will coordinate to create a 
risk identification process for the manual adjustment to customer bonus points. The risk 
identification process will include processes to query and compare customer's bonus points pre 
and post drawing to ensure points are removed or retained based on their participation in the 
bonus point program for that drawing and their drawing status. 


Recommendation #3 
We recommend the Department of Fish, Wildlife & Parks: 


A. Implement an on-going strategy to improve the clarity and consistency of bonus point 
information available to the public, and l 


Concur - The Licensing Bureau, Communication and Education Division and Technology Services 
Division staff will coordinate to create a statewide outreach strategy to better information 
FWP's customers regarding bonus points. Including, the purpose and benefit of purchasing and 
using bonus points from a customer perspective. 


B. Distribute new bonus point information to public facing FWP staff likely to interact with hunter 
and anglers. 


Concur - The Licensing Bureau and Communication and Education Division staff will coordinate 
to create statewide educational materials to better inform FWP staff regarding bonus points. 
This will allow public-facing staff to provide improved customer services as it relates to bonus 
points. 


Dustin Temple, Acting Director 
Department of Fish, Wildlife and Parks 


cc: Robin Graham, Operations and Financial Services Division Administrator 
Jessica Plunkett, Technology Services Division Administrator 
Emily Cooper, Licensing Bureau Chief 


